Probabilistic Modeling

A probabilistic model is a parametrized joint distribution over variables.

P (x 1, \dots, x n, y 1, \dots, y n ∣ ∣ θ)

$P\left(\left.x_1,\dots,x_n,y_1,\dots,y_n\right|\theta\right)$

Data: $x_1,x_2,\dots,x_n$
Latent variables: $y_1,y_2,\dots,y_n$
Parameter: $\theta$

Inference

P (y 1, \dots, y n ∣ ∣ x 1, \dots, x n, θ) = P ( x 1 , \dots , x n , y 1 , \dots , y n ∣ ∣ θ ) P ( x 1 , \dots , x n ∣ ∣ θ )

$P\left(\left.y_1,\dots,y_n\right | x_1,\dots,x_n,\theta\right) = \frac{P\left(\left.x_1,\dots,x_n,y_1,\dots,y_n\right|\theta\right)} { P\left(\left.x_1,\dots,x_n\right|\theta\right)}$
Learning
(Maximun Likelyhood)

θ = argmax θ P (x 1, x 2, \dots, x n ∣ ∣ θ)

$\theta=\underset{\theta}{\operatorname {argmax}}\ P\left(\left.x_1,x_2,\dots,x_n\right|\theta\right)$
Prediction

P (x n + 1, y n + 1 ∣ ∣ x 1, \dots, x n, θ)

$P\left(\left.x_{n+1},y_{n+1}\right|x_1,\dots,x_n,\theta\right)$
Classification

argmax c P (x n + 1 ∣ ∣ θ c)

$\underset{c}{\operatorname {argmax}}\ P \left(\left.x_{n+1}\right|θ^c\right)$

Bayesian Modeling

Prior distribution

P (θ)

$P(\theta)$
Posterior distribution

P (y 1, \dots, y n, θ ∣ ∣ x 1, \dots, x n) = P ( x 1 , \dots , x n , y 1 , \dots , y n ∣ ∣ θ ) P ( θ ) P ( x 1 , \dots , x n )

$P\left(\left.y_1,\dots,y_n,\theta\right | x_1,\dots,x_n\right) = \frac{P(\left.x_1,\dots,x_n,y_1,\dots,y_n\right|\theta)P(\theta)} { P\left(x_1,\dots,x_n\right)}$
Prediction

P (x n + 1 ∣ ∣ x 1, \dots, x n) = \int P (x n + 1 | θ) P (θ | x 1, \dots, x n) d θ

$P\left(\left.x_{n+1}\right|x_1,\dots,x_n \right)=\int P(x_{n+1}|\theta) P(\theta|x_1,\dots,x_n)\mathrm d \theta$